Skip to content

Set override_path = true in hosts.toml#10765

Open
rahulbabu95 wants to merge 1 commit into
aws:mainfrom
rahulbabu95:rahulgab/fix-hosts-toml-double-v2
Open

Set override_path = true in hosts.toml#10765
rahulbabu95 wants to merge 1 commit into
aws:mainfrom
rahulbabu95:rahulgab/fix-hosts-toml-double-v2

Conversation

@rahulbabu95
Copy link
Copy Markdown
Member

@rahulbabu95 rahulbabu95 commented May 11, 2026 •
edited
Loading

Issue #, if available:
#10212 introduced a regression while migrating the registry mirror config from config.toml (registry.mirrors endpoint format) to hosts.toml format. When oci namespaces are configured, ToAPIEndpoints() prepends /v2/ to t
he namespace path in the host URL. In the old config.toml format this was used as-is. In hosts.toml format, containerd's parser auto-appends an additional /v2/ to the host path, causing a double /v2/ in requests: /v2/< namespace>/v2/<repo>/manifests/<tag> resulting in 404.

Description of changes:
Add override_path = true to namespaced hosts.toml entries. This instructs containerd to use the host path as is without auto-appending /v2/. For base mirror entry (no path) we always append /v2/ for direct repo references so that override_path can be set unconditionally safely.

Testing (if applicable):
Pre-fix (setup a test ns rahul-tst in our CI registry and validate the image pull URIs from the node):
Hosts.toml:

server = "https://public.ecr.aws"

[host."https://10.80.148.51:443/v2/rahul-test"]
  capabilities = ["pull", "resolve"]
  ca = "/etc/containerd/certs.d/10.80.148.51:443/ca.crt"
  [host."https://10.80.148.51:443/v2/rahul-test".header]
    authorization = "Basic YWRtaW46eENnd3hqN0Y="

Image pull logs:
May 08 23:02:02 containerd[1404]: level=debug msg="fetch response received" response.status="404 Not Found" url="https://10.80.148.51:443/v2/rahul-test/v2/eks/cilium/cilium/manifests/v1.19.1-0?ns=public.ecr.aws"

Post-fix:
Hosts.toml:

server = "https://public.ecr.aws"

[host."https://10.80.148.51:443/v2/rahul-test"]
  capabilities = ["pull", "resolve"]
  override_path = true
  ca = "/etc/containerd/certs.d/10.80.148.51:443/ca.crt"
  [host."https://10.80.148.51:443/v2/rahul-test".header]
    authorization = "Basic YWRtaW46eENnd3hqN0Y="

response.status="200 OK" content-type=application/vnd.oci.image.manifest.v1+json url="https://10.80.148.51:443/v2/rahul-test/eks/cilium/cilium/manifests/v1.19.1-0?ns=public.ecr.aws"

Documentation added/planned (if applicable):

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@eks-distro-bot eks-distro-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label May 11, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented May 11, 2026 •
edited
Loading

Codecov Report

❌ Patch coverage is 60.00000% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.31%. Comparing base (c8c965a) to head (bd4a7c1).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
internal/test/registrymirror.go 0.00% 8 Missing ⚠️

❌ Your patch check has failed because the patch coverage (60.00%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files 
@@           Coverage Diff           @@
##             main   #10765   +/-   ##
=======================================
  Coverage   72.30%   72.31%           
=======================================
  Files         608      608           
  Lines       39388    39402   +14     
=======================================
+ Hits        28481    28493   +12     
- Misses       9173     9176    +3     
+ Partials     1734     1733    -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rahulbabu95 rahulbabu95 requested a review from snarkychef May 11, 2026 19:48
@rahulbabu95
Fix registry mirror hosts.toml double /v2/ regression from containerd…
bd4a7c1 
… v2 migration

URLs are used as-is) to hosts.toml format (where containerd auto-appends
/v2/ to the host path). Since ToAPIEndpoints() already prepends /v2/ before
the namespace, containerd produced a broken double /v2/ in requests:
/v2/<namespace>/v2/<repo>/manifests/<tag> → 404.

Fix: Always set override_path = true in hosts.toml entries so containerd
uses the host path as-is without auto-appending /v2/. Also update
ToAPIEndpoint to always produce a /v2 path prefix (even for pathless URLs)
ensuring all entries have a well-formed OCI API root.

Signed-off-by: Rahul Ganesh <rahulgab@amazon.com>
@rahulbabu95 rahulbabu95 force-pushed the rahulgab/fix-hosts-toml-double-v2 branch from bcd55c6 to bd4a7c1 Compare May 12, 2026 23:52
@eks-distro-bot eks-distro-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 12, 2026
@rahulbabu95
Copy link
Copy Markdown
Member Author

rahulbabu95 commented May 21, 2026

/cherrypick release-0.25

@eks-distro-pr-bot
Copy link
Copy Markdown
Contributor

eks-distro-pr-bot commented May 21, 2026

@rahulbabu95: once the present PR merges, I will cherry-pick it on top of release-0.25 in a new PR and assign it to you.

Details

In response to this:

/cherrypick release-0.25

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@rahulbabu95 rahulbabu95 changed the title Set override_path = true to namespaced hosts.toml entries in hosts.toml Set override_path = true in hosts.toml May 21, 2026
@rahulbabu95
Copy link
Copy Markdown
Member Author

rahulbabu95 commented May 21, 2026

/cherrypick release-0.24

@eks-distro-pr-bot
Copy link
Copy Markdown
Contributor

eks-distro-pr-bot commented May 21, 2026

@rahulbabu95: once the present PR merges, I will cherry-pick it on top of release-0.24 in a new PR and assign it to you.

Details

In response to this:

/cherrypick release-0.24

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@snarkychef
Copy link
Copy Markdown
Member

snarkychef commented May 22, 2026

/lgtm
/approve

@eks-distro-bot
Copy link
Copy Markdown
Collaborator

eks-distro-bot commented May 22, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: snarkychef

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@snarkychef snarkychef Awaiting requested review from snarkychef

Assignees

@snarkychef snarkychef

Labels

approved lgtm size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@rahulbabu95 @eks-distro-pr-bot @snarkychef @eks-distro-bot